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< \\ The Problem 

' ■■■ §. ^ 

The application of basic scientific-technological methodology to 
learning systems is not a recent phenomenon. Having its genesis in the 
programmed instruction movement, what we now call instructional technology, 
instructional system development, or the systems approach has been applied 
to a vast number. of military, industrial, and educational training develop- 
ment efforts (Saettler, 1968). The behaviorally-oriented systems approach 
to instruction prescribes the following interdependent operations as a 
means of achieving learning effectiveness: (1) Selecting responses to be 
practice^ on the basis of behavioral analysis of learning tasks; (2) deriv- 
ing objectives from the behavioral analysis which describe the behavior at 

the end of training; (3) developing test items which sample the behavioral 

i 

domain specified in the objectives; (4) selecting media which have instruc- 
tionally relevant stimulus display and student response capabilities; (5) 
field testing the material on a representative sample of the target popu- 
lation; land (6) revising the material on the basis of the tryout results 
so that training objectives will be achieved. 

Research on components of the .systems approach and system development 
state-of-the-art, theory and practice have been reported elsewhere (Baker J 
1973), but a review of the literature fdiled to uncover research on the 
relative effectiveness of systematic development per se. 

A number of formal, rationally derived procedural models have been 
developed which operationalize the above steps and thus guide the develop- 
ment of instructional material (Briggs , 1970; Butler, 1967; Drumheller, 
1971; Gropper, 1971; Schumacher & Glasgow, 1974; Smith, 1966; Wong & 
Raulerson, 1974). Typically, the models specify step-by-step procedures, 



Which if followed will presumably lead to instructional materials which, 

* ? 

result in instructional materials of a higher quality thanJJiose developed 



using a le^s systematic or 
costs are higher than when 



an unsystematic approach. Whilj2 developmental 

less formal approaches are used, the argument 

•! 

usually put forth in its defense is that it is more cost effective since 
more students achieve *a higher standard of performance. Yet, the sheer 
Complexity* of some of the developmental models cast doubt on this assump-v 
tion. For example, the Northwest Regional Educational Laboratory includes 
37 steps in product development and installation, -31 of which are directly 
related to development (Bak^r, 1973). The American Institutes for Research 
(Gropper, 1971) has published a model which contains 37 macro steps, with 

, ' :i 

a greater number of micro sti|eps subsumed under these. Therefore, one must 

. * ii 
logically ask if the scope otf the developmental undertaking is worth the 

payoff. < , 

Anyone who has attempted systematic development knows that, if done 
properly, it k is a time-consuming task, and because time means money, it 
can also be costly. However, in some fields wh^re the" demand for trained, 
qualified personnel exceeds supply, the instructional technologist is under 
tremendous pressures to produce high quality training within very con- 
stricted time frames, and often under budgets established by 
managers who have little understanding pf the instructional ^development , 
requirements . ^ Therefore, given the kind's of pressures on instructional 
designers in applied learning, it is imperative that ways of reducing costs 
while maintaining professional standards be explored. 
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Research Needs 

While it is the conventional wisdom that systematic development is 
cost effective, no empirical test of the validity of this assumption has 
been conducted. The cost effectiveness of using different teaching methods 
and programs has been evaluated within some elementary, secondary, univer- 
sity, and professional schools (Levin, 1975), but cost effectiveness of 
developing self-instructional materials through different instructional 
development approaches has not been compared. That is, for a given set of 
instructional materials, datp, do not exist which show comparison of test 
scores, comparison of costs associated with the application of different 
approaches to producing, the instructional material, and comparison of 
't imes required to develop the \nstructional material. While there seems 
to be a surfeit of guides for employing the systems approach to develop 
instructional programs, the research in support of such practice in develop- 
ment is yet to be found. * 

Variables likely to influence cost ef f ectivenes^ include: 

1. The degree to which all steps in systematic approaches 
are formally applied. (The manner in which developmental 
steps are executed . ) 

2. The complexity^pf^the learning tasks which are the 

object of/inst ruction. , / 

3. The orientation of the organization and personnel / 
responsible for developing instruction. j 

The Execution of Steps in Procedural Models , "j 

^^Strict, formal application of steps delineated by- instructional design 
models may be more critical in the development process than others. Formal 
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analyses or procedures are important when they force the instructional 

technologist to come to grips with issues in more substantive ways than 

he might otherwise; structures the technologist's behavior so that all 

relevant factors are more likely to be' considered than overlooked; and 

provides a tool which reduces unproductive behavior, and thus allows the 

instructional technologist to more quickly arrive at an effective solution. 

However, for certain steps the same results may be achieved from less rigid, 

• * 

heuristic approaches which do not rely on the kind of highly proceduralized , 
formal analysis specified in many developmental models. Therefore, there 
exists a need to determine the level of development complexity necessary to 
, ; priroduce cos t-effective ins true tilon. 

Applied Science Associates is participating in a study being conducted 
by the National Library of Medicine (Note 1). In the study, ASA will devel- 
op a set of instructional materials using a semi-formal approach. The proj- 
ect is designed to investigate alternative levels of instructional develop- 
ment complexity, how these different levels affect spending, varying degrees 
of student achievement associated with each level, and the* time required to 
devel©p 4 the materials. The three levels of instructional development com- 
plexity are: (1) A formal approach in which systematic ^procedures are 
strictly and formally applied; (2) a semi-jLarmal approach in which only 

A:o 



selected steps 



none of the steps typically employed in systemat 



In using 



the semi-formal approach, we will conduct the following in a 



are strictly applied; and (3) an 




ach in which 
elopment are used. 



structured fashion: analyze learning tasks, formulate objectives, and re- 
vise the material. We believe the ■ formal approach is critical in conducting 



Note 1: The research upon which this publication is based is being per- 
formed pursuant to Contract No«. N01-LM-6-4731 with the National 
Library of Medicine, National Institutes of Health, Department 
of Health, Education and iWelfare. 



a learning analysis because the conditions -'fro \e established for effective- 
learning depend ,to an important extent ■ -np^' t t)jLe* identification of what class 
of human performance is to be learned ^(-CJagne, 1965). Precisely stated 
objectives derived from the learning analysis are important because they 

aid the instructional designer in planning his instructional package and 

. $ - 

to later evaluate its effectiveness. * 

Finally, formal revision strategies are critical when they focus on 
types of errors, rather -than error rate , and systematically take into 
account the probable reasons for failure of strategies prescribed earlier, 
and address themselves to why the treatments -so obviously failed. If re- 
vision proceeds on an ad hoc basis instead, then the tryout and revision 
cycle is likely to be extended until the instructional designer "hits 11 on 
the right strategy , w thus extending developmental t>ime and costs. 

How we intend to avoid fortnal analysis in some steps, while perform- 
ing it in others probably needs clarification. In the systems approach to 
instructional development, the output of each step serves as the input to 
the next step. For example, the foundations for test -construction are set 
when behavioral objectives are written. However, test development may fol- 
low one of two divergent paths: informal and formal. In the informal 
approach, test development^Ee^gin^ when the test writer attempts to produce 
items which incorporate the response condition specified in the objective. 
Thus, an objec/tive which calls for the student to select a response should, 
at .best, h^e with it items where the learner is actually making a choice. 
The stimulus ends of items are ba,sed on the writer's^ subjective decisions 
about stimulus boundaries and variability. He builds a test which he 
judges to. be of reasonable length to allow generalization about the specif- 
ic types* of task's an individual ^can perform. For some types of performances, 

• 7 
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1 the relevant domain erf test items can number into th£ th^usand^ -but deci- 
l sions about test length are usually tempered by practical considerations 

rather than strict consideration of variables suph as ta&k variability. 
• Thus, he follows no formalized procedures, but rather develops items which, 
in his best judgment are appropriate for the objective as written. In sum, 
while based on systematically derived objectives, most decisions^egarding 
content limits, distractor domains, formats, length, etc. are, in fact, 

jlarbitrary (i.e./ informal). 

1 1 /I 

1; In contr^fet to the above approach, a theory of domain-referenced test 

i 

construct ton Jhas evolved which specifies highly formal, systematic proce 

\ 

K d\xres for constructing tests. Baker (1974), for example,' identifies six 
ajteps of dojfnaia-referenced ,test specif ications including rules of contfent 
eligible fpr test inclusion,, rules fo\r inclusion of wrong-answer>#tterna- . 
tives in rjtultiple choice items, and rijile^ in sampling the itemjpools./ We 
propose^ tib avoid -the type of highly prjoceduralized, systematic analysis in 
ich a I 



which a Ijarge number of test items ar& generated .according to given /rules, 



and then 



randomly sampled to constitute the test. Rather, we will /follow 



the informal, unstructured approach described in the above paragraph. 

\ f \ , If 

Also, it is not our intention to conduct a formal field test /in which 

. / * 

a randomly selected sample of the target population is* tested on /the program 
in an operational setting.' Nor do we intend to systematically repeat a 
tryout and revision cycle until a predetermined criterion is reached. Many 
instructional development models have carefully prescribed procedures for 

field testing materials. The Briggs 1 model (1970), for example, tecommends 

I 

eight 'separate activities for formative evaluation alone. We/ intend to 

" 0 // 

ft 

avoid any approximation of any standardized or formalized procedures. 
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Yet, inherent in our decision to formally implement the revision step, 

is the need to collect tryout data. We intend to collect this data by 

having students try out the materials as they are developed. Tryouts.will 

*be conducted whenever the project staff judge the materials to be in a 

reasonable form, and no consideration will be given to how well subjects 

represent the population. How many students will take part in the tryout,^ 

whether all students will encounter the material collectively ^r not, and 

whether all students will encounter the same version of" the material will 

be determined by practical considerations, rather than formal requirements. 

* 

The steps in a systems approach are interdependent, and this fact has 
important implications for the study. For example, it. can be argued that 
the ^formulation of objectives is inherent to the development of an objective 
test. Whether the objectives are formally constructed and syste|nat ically 
written down or simply a set of ideas in the mind of the test developer, 
they We, nevertheless, established. In'the first case, they, are formalized 
according to some set of rules befofe the test development phase; in the 
second case, they evolve heuristically as the test is being developed. 
Therefore, the difference between an informal and formal approach is not 
whether objectives were developed, rather the difference is whether they 
were formalized. This- line of reasoning^can te extended to tfhe tryout and 
revision -steps. ' We cannot logically revise material which has not bger^ 
tried out. s But we can avoid any formalized procedures, and simply an^ ^ 
informally, gather information, while strictly and rigorously analyzing 

the* data. * * ' ' 

This initial effort will make an important contribution toward a bet- 
ter understanding, of "th^e practical , role played by formal developmental models. 
There are, of course other factors which may influence cost-effectiveness. 

. 9 ■ 
7 



The Difficulty of the Learpqirg Ta sk ^ <<ta ^ 
While a learning analysis is a critical, step in the development process, 
its relative effectiveness may differ depending on the nature of the learning 
task. If a learning task is judged to be easy, for example, it may be cost 
effective to allow the ins tructi'onal ^technologist to set up objectives 
(omit'ting a formal analysis) and proceed with designing instructional strat- 
egies. It has been our experience that often during: the process of develop- 
ing x lessQn plans, the programmed text, or other instructional material, the 
designer heuristically uncovers critical task variables that influence 
learning outcomec^^^des igns instruction accordingly. 

The implication of this is that the task analysis activities are, in 

** * ■ , 

fact, taking place, but simultaneously w/Lth development rather than preced- 
ing it. The difference is that with this approach, criterion objectives 
may be achieved without having performed the task analysis in any formal 
way, and much unnecessar| work documenting task analysis results wil^have 
been avoided. 

When the criterion objectives are not achieved, the kind of error 
analysis previously discussed will identify those elements of the learning 
task requiring systematic analysis. If revision strategies are the mirror 
image of learning analysis, the previously omitted learning analysis will 
be performed post hoc. But now, it wijrl focus only on those aspects of the 
task where learning has so obviously failed. Thus., if a substantial part 
of the instruction remains intact, analysis will be less than full scale. 
Consequently, less time and money will have been expended. 

In sum, task difficulty is a variable which should be evaluated in 
addition to levels of developmental complexity when exploring ways to re- 
duce cost while maintaining professional standards. It may be that for 



y&- . 10 



less complex tasks, front-end analytic developmental activities can be 
precluded, but the same activities may be imperative for moire cromplex fotnjs ^ 
of learning because of the probability that without it, instruction will 
seriously miss tile mark and substantive costly revision ejisue. 

Orientation of Organization and Personnel 

" : ~ 'f 

One of the dangers in the delineation and dissemination of stages of 

critical events or steps in development is that they may be formally employed 1 , 
but incompletely understood. For example, in an assessment of needs, (Schu- 
macher, et al , 1972) personnel responsible for designing instructional systems 

0 

and developing training courses reported considerable difficulty with all ' 
of the major elements pf the process although they had been using it for 
some time. Those activities identified as the most troublesome included 
the two areas of greatest technological advancement: task analyses and 
behavioral objectives. Neither practice is new. With respect to object ives,, 
virtually all educational and training personnel who plan, develop, approve, 
administer, or manage instruction and its supporting material are aware of 
thfe criticality of precisely stated objectives. Yet, despite all the guid- 
ance which has been published, people apparently are still failing to under- 
stand and apply the process in any meaningful way. 

NIE f s Group on School Capacity for Problem Solving reports that^'many 
educational innovations- fail because they are* implemented poorly or not at 
all (Tucker, 1975). NIE reports that with respect to Federal programs, neithe 
the way in which Jthey were framed and managed, nor the size of the funding 
accounted for most of the variance in implementation success. What really 
counted was the orientation of those receiving the funding, and certain 
organizational an & managerial characteristics of the project and institution. 
It is no wonder that so many innovations produce "no significant dif ferences 
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Results from research on systematic development are likely to show 
tjhe same results unless funding sources develop mechanisjns for avoiding * 
the all *too common situation ia which a practitioner's ability to use the 
jargon of the field is misconstrued as evidence of his ability to meaning- 
fully implement a systems approach. Developers participating In research 
must have a good understanding of systems analyses methodologies and pro- 
cedures as problem solving tools, rather than a series of step's which, if 
mechanically followed, wiH p'roduce^ cost-effective outcomes. 

Therefore, if organizational factors 'are ^impqrtant determiriants of 
the success of innovations, and the quality' of educational outcomes, then 



we must begin now to identify those organizations^ where innovations fr&ve 
been successful, and describe the methods and capa<^4€ies of their organiza- 
tions in order to develop and train future professionals in their image. e 



Summary '" 



Clearly, t;hete has been no lack of literature describing instructional 

- / / 

system practices and procedures. However, pressures exerted by the nation's 
political, professional, researcl^, and economic subgroups to train more 
qualified personnel faster in fietLds such as the health prof essions , A'ave 

created a neld to examine ways oflreducing costs while maintaing profes- 

1 '• •■ »* I ■ * 

siotial standards '.and creating a mo\re efficient instructional process 
Three of the many factors that must' jbe explored in order to intelligent" 



assess alternative strategies were lcfl^cussed. 
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